Skip to content

fix(vector): a truncated FT.CREATE no longer aborts the whole process (#681) - #682

Merged
TinDang97 merged 1 commit into
mainfrom
fix/ftcreate-truncated-panic-681
Aug 23, 2026
Merged

fix(vector): a truncated FT.CREATE no longer aborts the whole process (#681)#682
TinDang97 merged 1 commit into
mainfrom
fix/ftcreate-truncated-panic-681

Conversation

@TinDang97

@TinDang97 TinDang97 commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Closes #681.

The bug

$ redis-cli FT.CREATE tidx ON HASH PREFIX 1 d: SCHEMA v VECTOR HNSW
Error: Server closed the connection
$ redis-cli PING
Could not connect to Redis: Connection refused
thread 'shard-0' panicked at src/command/vector_search/ft_create.rs:550:39:
index out of bounds: the len is 10 but the index is 10
FATAL: thread 'shard-0' panicked; aborting the whole process rather than
serving with a dead shard or a dead cluster control plane

parse_vector_field_params bounds-checks the algorithm keyword, advances past
it, then reads the parameter count with no second check. When HNSW is the
last argument, *pos becomes args.len(). The panic is on a shard thread and
moon escalates that to a process abort by design, so the blast radius is the
whole server — every database, every other connection. No auth, no large
payload, one short line.

Present on main since the file was written (#27).

Scope was measured, not assumed

My first read was that every *pos += 1; args[*pos] pair in the function was
its own crash — about a dozen. That was wrong. The parameter loop guards both
ends (*pos + 1 < param_end && *pos + 1 < args.len()), so every value read
is safe; only the count read was unguarded.

Six truncation shapes, each against a freshly spawned server whose listener PID
was checked against the process I started:

Truncated at Result
... VECTOR HNSW DEADft_create.rs:550:39
... HNSW 6 TYPE alive
... HNSW 6 TYPE FLOAT32 DIM alive
... HNSW 6 TYPE FLOAT32 DIM 4 DISTANCE_METRIC alive
... HNSW 6 TYPE FLOAT32 DIM 4 M alive
well-formed control alive

A second test pins those seven neighbours as already-bounded, so a future edit
to the loop condition cannot quietly reopen the hole one keyword further in.

The fix

One bounds check returning ERR invalid param count — the error an
unparseable count already produced, so the two ways of failing to supply a
count are indistinguishable to a client. There is no redis oracle for the
string: the redis-server checked against has no query engine, so FT.CREATE
is unknown command there.

The fuzz target took three designs

FT.CREATE argument parsing had no fuzz target for its whole life, which
is how a one-line remote crash survived in it. Each design was rejected by
measurement against a deliberately un-fixed parser, not by inspection:

  1. Fully-arbitrary argv — 1.2M execs, found nothing. ft_create demands
    the preamble idx ON HASH PREFIX 1 d: SCHEMA v VECTOR before the vector
    parser is reached, and random mutation does not synthesize a nine-keyword
    sequence. It was fuzzing the preamble and never getting past it: builds,
    runs, reports clean, cannot see the bug it exists for.
  2. Skeleton + byte-level tail — 871K execs, still nothing. Reaching the
    parser is not enough; the crash needs the literal ASCII HNSW in argv, and
    inventing a specific four-byte string by mutation is a 2^32 search.
  3. Skeleton + one input byte selecting one argv element from the vocabulary
    the parser branches on
    — finds the panic and writes a reproducer.

Verified in both directions:

  • against the un-fixed parser it crashes at ft_create.rs:561 with
    index out of bounds: the len is 10 but the index is 10 and saves an
    artifact;
  • that same artifact executes clean against the fix (Executed ... in 46 ms).

Listed in both matrices in fuzz.yml — a target that exists but is not
listed never runs (#576) — and confirmed not caught by the bare-fuzz
gitignore trap.

Verification

  • 4/4 new unit tests. The first was proven red for the right reason: it
    failed with the panic itself, not an assertion mismatch.
  • End-to-end on a live server, listener PID checked: the crashing command
    answers ERR invalid param count, the server stays up, zero panics in the
    log, FLAT keeps its own distinct message, and a well-formed FT.CREATE
    still returns OK with an index queryable via FT.INFO.
  • cargo fmt --check, clippy --all-targets -D warnings on both the
    default and runtime-tokio feature sets.

Summary by CodeRabbit

  • Bug Fixes

    • Fixed a server crash when FT.CREATE commands contain truncated or incomplete vector-search arguments.
    • Invalid parameter counts now return a clear ERR invalid param count error.
    • Improved handling of malformed and incomplete vector index configurations.
  • Tests

    • Added expanded validation coverage for truncated, invalid, and boundary-case FT.CREATE arguments.
    • Added fuzz testing to improve parser robustness.

…#681)

`FT.CREATE idx ON HASH PREFIX 1 d: SCHEMA v VECTOR HNSW` -- argv cut off
right after the algorithm keyword -- read one past the end of `args` and
panicked. The panic ran on a shard thread, and moon deliberately escalates a
shard panic to a whole-process abort rather than serve on with a dead shard,
so one short line from any client took the server down: every database,
every other connection. No auth, no large payload.

The fix is one bounds check returning `ERR invalid param count`, the error an
unparseable count already produced, so the two ways of failing to supply a
count are indistinguishable to a client. There is no redis oracle for the
string -- the redis-server checked against has no query engine, so FT.CREATE
is `unknown command` there.

Scope was measured, not assumed. The first guess was that every
`*pos += 1; args[*pos]` pair in the function was a separate crash; six
truncation shapes probed against freshly spawned, listener-PID-checked
servers said otherwise. The parameter loop guards both ends
(`*pos + 1 < param_end && *pos + 1 < args.len()`), so every value read was
already safe -- the count read was the one unguarded site. A second test
pins those seven neighbours so a future edit to the loop condition cannot
quietly reopen the hole one keyword further in.

FT.CREATE argument parsing had no fuzz target for its whole life, which is
how a one-line remote crash survived in it. The new target took three designs,
each rejected by measurement against a deliberately un-fixed parser rather
than by inspection:

  1. Fully-arbitrary argv -- 1.2M execs, found nothing. `ft_create` demands
     the preamble `idx ON HASH PREFIX 1 d: SCHEMA v VECTOR` before the vector
     parser is reached; random mutation does not synthesize a nine-keyword
     sequence, so it fuzzed the preamble and never got past it.
  2. Valid skeleton + byte-level tail -- 871K execs, still nothing. Reaching
     the parser is not enough: the crash needs the literal ASCII `HNSW` in
     argv, and inventing a specific four-byte string by mutation is a 2^32
     search.
  3. Skeleton + one input byte selecting one argv element from the vocabulary
     the parser branches on -- finds the panic, writes a reproducer artifact.

Verified in both directions: the target crashes the un-fixed parser at
ft_create.rs:561 with `index out of bounds: the len is 10 but the index is
10`, and that same saved artifact executes clean against the fix. Listed in
BOTH matrices in fuzz.yml -- a target that exists but is not listed never
runs (#576).

Also verified end-to-end on a live server with the listener PID checked
against the spawned process: the crashing command answers `ERR invalid param
count`, the server stays up, zero panics in the log, and a well-formed
FT.CREATE still returns OK with a queryable index.

Gates: cargo fmt --check, clippy --all-targets -D warnings on default AND
runtime-tokio feature sets, 4/4 new unit tests.

author: Tin Dang
@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Aug 23, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 522c2839-77db-4386-9df3-eeda03a1bf4b

📥 Commits

Reviewing files that changed from the base of the PR and between a3e6e6f and f25eb46.

📒 Files selected for processing (5)
  • .github/workflows/fuzz.yml
  • CHANGELOG.md
  • fuzz/Cargo.toml
  • fuzz/fuzz_targets/ft_create_args.rs
  • src/command/vector_search/ft_create.rs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The FT.CREATE vector-field parser now rejects a missing HNSW parameter count instead of panicking. Unit tests cover truncated inputs, and a new ft_create_args fuzz target runs in pull-request and nightly workflows.

Changes

FT.CREATE parser safety

Layer / File(s) Summary
HNSW parameter validation
src/command/vector_search/ft_create.rs
The parser checks for a missing HNSW parameter count and returns ERR invalid param count. Tests cover truncated parameter lists, invalid counts, and algorithm errors.
Parser fuzz coverage and CI wiring
fuzz/fuzz_targets/ft_create_args.rs, fuzz/Cargo.toml, .github/workflows/fuzz.yml, CHANGELOG.md
The new fuzz target generates valid and malformed FT.CREATE arguments, initializes fresh stores, and invokes ft_create. The target is registered in both fuzzing matrices and documented in the changelog.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to f25eb

The PR prevents truncated FT.CREATE commands from terminating the server while preserving valid command behavior; no actionable merge-blocking risk remains after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant libFuzzer
  participant ft_create
  participant vector_store
  participant text_store
  libFuzzer->>ft_create: generated FT.CREATE argument frames
  ft_create->>vector_store: initialize vector store
  ft_create->>text_store: initialize text store
  ft_create-->>libFuzzer: process survival or failure
Loading

Suggested reviewers: pilotspacex-byte

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the fix for truncated FT.CREATE commands causing a process abort.
Description check ✅ Passed The description provides detailed bug context, fix scope, testing results, and fuzzing coverage, although it omits the template headings and some checklist items.
Linked Issues check ✅ Passed The changes address issue #681 by preventing the panic, returning ERR invalid param count, adding regression tests, and adding fuzz coverage.
Out of Scope Changes check ✅ Passed The parser fix, tests, changelog entry, fuzz target, and workflow updates are directly related to issue #681.
Docstring Coverage ✅ Passed Docstring check was indeterminate for this PR — some files could not be analyzed in time. Not blocking.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/ftcreate-truncated-panic-681

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@TinDang97
TinDang97 merged commit 9516931 into main Aug 23, 2026
19 checks passed
TinDang97 added a commit that referenced this pull request Aug 23, 2026
…tals (#679) (#684)

The suite CLAUDE.md points at for every new command had never printed a
result. It died partway through, silently, and the operator saw a truncated
log instead of a summary. It now finishes: 504 rows, 478 passing.

The 26 remaining failures are pre-existing and are filed as #683 -- they
became visible for the first time because the run now reaches them. Nothing
here weakens an assertion to get a green.

Every abort was the same bash shape: a command whose non-zero status
`set -euo pipefail` converts into a silent exit. grep exits 1 when it matches
nothing, lsof exits 1 when a port is free, pkill exits 1 when nothing matched,
and a shell function whose last statement is a false `if` returns 1 as well.
Because that failure prints nothing at all, each instance hid the next --
which is why this took several rounds, and why two of the guards below are
for aborts introduced by earlier ones in this same commit.

Guarded:
  * 26 command substitutions ending in grep. The reported NUMERIC-07 site was
    one of a class, so the class is fixed rather than the instance.
  * 6 raw redis-cli pipelines and all four client wrappers, so a dead server
    yields failing rows plus a summary instead of a truncated log. Nothing in
    the file branches on those wrappers' exit status (checked: no `if mcli`
    and no `mcli ... &&` anywhere), so `|| true` costs no signal.
  * `cargo build ... 2>/dev/null`, which threw away the reason a build failed
    and left the log reading "Building moon..." and nothing else.
  * the cleanup trap, which returned its last kill's status rather than the
    script's -- reporting a clean run as a failure.

Three defects that produced wrong results rather than aborts:

  * `grep -Pzo "(?s)A.*B"` at 13 call sites is GNU-only. On a macOS host grep
    is ugrep, which rejects -P and exits 2; since those rows compare output
    rather than status, that 2 was being reported as moon's answer. Replaced
    with a portable spans() helper.
  * No --dir, so moon treated the CWD as its data dir: the suite wrote
    appendonlydir/ and moon.lock into the repo root and reloaded the previous
    run's FT index definitions, so a second run failed with "Index already
    exists". Each run now gets a fresh mktemp dir, removed on exit.
  * No port pre-flight. A leftover server from an unrelated run answers and
    every row silently compares against it. Not hypothetical: it produced a
    full run of MOONERR diskfull failures traced to another session's moon on
    the port. The suite now refuses to start on an occupied port and names the
    holder.

The `FT.CREATE ... VECTOR FLAT` row expected OK, but moon has only ever
implemented HNSW (ERR expected HNSW algorithm, in ft_create.rs since #27), so
it had failed from the day it was written and took four dependent rows with
it. It now builds an HNSW index, and the FLAT gap is asserted explicitly
instead of hiding inside a row that expected success.

Added a regression row for #681 asserting the server is still alive after a
truncated FT.CREATE. Proven in both directions: it fails against a pre-#682
binary and passes after.

CLAUDE.md's "190 tests" was stale by more than half.

Fixes #679
Refs #681, #682, #683

author: Tin Dang
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A truncated FT.CREATE (... VECTOR HNSW with nothing after) panics the shard thread and aborts the whole process

1 participant